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Abstract 

The purpose of this paper is to measure and compare the effectiveness of a Web-based 
course delivery system to a traditional course delivery system. The results indicate that a 
web-based course is effective and equivalent to a traditional classroom environment. As 
with the implementation of all new technologies, there are some pros and cons that 
should be considered. The significant pro is the element of convenience which eliminates 
the constrictive boundaries of space and time. The most notable con involves the 
impersonal nature of the online environment. Overall, we found the web-based course 
delivery system to be very successful in terms of learning outcomes and student 
satisfaction. 
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Introduction 

There has been a proliferation of courses and programs that are available over the World 
Wide Web. This includes both distance education and resident student instruction. 
Traditional bricks and mortar courses have a long history of proving how effective they 
are through exams, student evaluations and student feedback. Yet, there has been a 
lingering debate in the literature and halls of academia as to whether online courses are as 
effective as traditional courses (Chen and Jones 2007; and Hiltz 1993). 

The authors wanted to investigate if there are differences in learning in a Web-based 
section of a course and learning in a bricks and mortar section of the same course. We 
also wanted to investigate student satisfaction between a Web-based section of a course 
and a bricks and mortar section of the same course. There has been some recent research 
completed in regard to these topics. 

Literature Review 

Across the literature we found that there were several common assessment issues in 
regards to web-based courses. These include learning outcomes, course satisfaction, and 
technical issues. The last 14 years has provided a wealth of information with inconsistent 
results and few conclusions as to the effectiveness of web-based courses versus 
traditional brick and mortar. Some studies have been so broad as to look at the impact of 
institutional mission, faculty development, and course development strategies (Evans, 
2001). Some studies even discussed potential barriers that faculty should be aware of 
when sharing information with students’ online (Suter and Kopp, 1998). The general 
focus of this discussion will look at learning outcomes, course satisfaction and technical 
issues, as this has been the primary discussion in the literature. 

Learning Outcomes 

When looking at learning outcomes, researchers have found very different results (Vat 
2006). They have identified trends, factors for success, and potential advantages and 
disadvantages associated with web learning (Hollenbeck et al. 2005). Vogt, Atwong and 
Fuller (2005) found that students in an advanced business communication course did 
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achieve a high level of proficiency and they did so equally in both traditional and online 
classes. Bata- Jones and Avery (2004) in research on nursing students who took a 
pharmacology course found that there were no significant differences between the mean 
exam scores of students enrolled in the web-based and traditional courses. In 2004, 
Kearns, Shoaf and Summey found that students in a Web-based second-degree bachelor 
of science (in Nursing) program scored significantly higher on the final examination and 
the comprehensive examination than did students in the traditional course. Buckley 
(2003) found that students in a web-based nutrition course received a lower mean course 
evaluation score than students in a traditional similar course. In 2002, Maki and Maki 
found that students in a Web-based course (psychology) learned more than students in a 
bricks and mortar section of the same course. Arbaugh and Duray (2002) compared two 
web-based MBA programs, one with some on-site meetings and the other totally online. 
They found that larger class sizes were negatively associated with perceived learning. 
Sankaran, Sankaran and Bui (2000) researched the amount learned in Web-based and 
bricks and mortar versions of an undergraduate business computer course. No significant 
difference in learning gain from pretest to the final exam was found. Wang and Newlin 
(2000) compared Web-based and bricks and mortar sections of a statistics course. They 
found that the bricks and mortar students scored higher on the final exam than did the 
Web-based students. Hiltz (1993) compared Web-based and bricks and mortar sections 
of several courses and found equal learning in the Web-based and bricks and mortar 
section in almost all courses and superior learning in the Web-based computer science 
course. 

Table 1 illustrates a summarization of these various studies. An analysis of this table 
indicates that in 7 of the 9 studies there was no difference in learning outcomes or that 
there were higher learning outcomes than in traditional courses. While there is no 
definitive conclusion from looking at these studies, it does suggest that there is potential 
for achieving comparative or even higher learning outcomes in web-based courses. 
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TABLE 1: Learning Outcomes Studies 


Authors 

Sample 

Outcomes 

Vogt, Atwong and Fuller 
(2005) 

Business Students 

No difference between Web and 
Brick n’ Mortar 

Beta-Jones and Avery (2004) 

Nursing students 

No difference between Web and 
Brick n’ Mortar 

Kearns, Shoaf & Summey 
(2004) 

Nursing (second Bachelors) 

Web students earned bigber 
scores on final exams and comps 
than traditional students 

Buckley (2003) 

Nutrition students 

Web students earned lower mean 
course evaluation scores than 
traditional students 

Maki and Maki (2002) 

Psychology students 

Web students learned more than 
traditional students 

Arbaugh and Duray (2002) 

MBA students 

Web students bad higher 
perceived learning than 
traditional students 

Sankaran, Sankaran, and Bui 
(2000) 

Business students 

No difference between Web and 
Brick n’ Mortar 

Wang and Newlin (2000) 

Business students 

Web students earned lower 
scores on the final exam than 
traditional students 

Hiltz (1993) 

Computer science students 

Web students learned more than 
traditional students 


Student Satisfaction 

When looking at level of student satisfaction in Web-based courses versus bricks and 
mortar courses results also varied in the studies we found. Chen and Jones (2007), 
Stanley (2006) and Arbaugh (2005) all find that satisfaction was similar to traditional 
course delivery systems. Yet, in 2004, Kearns, Shoef and Summey found less satisfaction 
for students in a Web-based second-degree bachelor of science (in Nursing) program than 
the traditional course method. Bata-Jones and Avery (2004) in research on nursing 
students who took a pharmacology course found that students in enrolled in the web- 
based course were more positive about their experience than those enrolled in the 
traditional course. In 2003, Buckley in her study on students in nutrition classes had 
much more neutral findings, where the students expressed both positive and negative 
aspects of the online instruction. In 2002, Wills found that students seemed to favor an 
online format for the teaching of graduate level nursing courses versus the traditional 
classroom environment. Maki et.al. (2000) found lower satisfaction in the Web-based 
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than in the bricks and mortar version of introductory psychology, a finding replicated by 
Maki and Maki (2002). Arbaugh and Duray (2002) in their investigation of Web-based 
MBA programs found that more experienced on-line students tended to be more satisfied 
with Web-based delivery mechanisms. They also found that smaller class sizes were 
positively associated with satisfaction. Wang and Newlin (2000) found fairly equivalent 
satisfaction. Hiltz (1993) found that satisfaction was high in the Web-based courses (but 
he did not compare Web-based vs. bricks and mortar). Table 2 provides a summary of 
these studies which illustrates that in 8 of the 1 1 studies satisfaction was the same or 
higher than traditional courses. While there appears to be some disparities between the 
studies, it looks like course satisfaction in web-based courses is favorable when 
compared to traditional courses. 


TABLE 2: Student Satisfaction Studies 


Authors 

Sample 

Outcomes 

Chen and Jones (2007) 

MBA students 

No difference between Web and 
Brick n’ Mortar 

Stanley (2006) 

Medical students 

No difference between Web and 
Brick n’ Mortar 

Arbaugh (2005) 

MBA students 

No difference between Web and 
Brick n’ Mortar 

Kearns, Shoaf & Summey 
(2004) 

Nursing (second 
Bachelors) 

Web students reported less 
satisfaction than traditional students 

Beta-Jones and Avery (2004) 

Nursing students 

Web students reported bigber 
satisfaction than traditional students 

Buckley (2003) 

Nutrition students 

No difference between Web and 
Brick n’ Mortar 

Wills (2002) 

Graduate Nursing 
students 

Web students reported higher 
satisfaction than traditional students 

Maki and Maki (2002) 

Psychology students 

Web students reported less 
satisfaction than traditional students 

Maki et al. (2000) 

Psychology students 

Web students reported less 
satisfaction than traditional students 

Arbaugh and Duray (2002) 

MBA students 

Web students reported bigber 
satisfaction than traditional students 

Wang and Newlin (2000) 

Business students 

No difference between Web and 
Brick n’ Mortar 
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Technical Issues 

One of the issues frequently facing faculty members teaching web-based courses is the 
level of technical skills on the part of the students (Su, 2005). Kaynama and Keesling 
(2000) found the variations in students’ technical skills constituted the most challenging 
task the instructor had to work through. This is a problem that was shared with many 
other educators (Martins and Kellermanns, 2004; and Siu and Chau, 1998). Although, 
this may not be considered an outcome variable, it may be important as a predictor of 
success. 

In summary, a majority of studies that investigated learning outcomes found that web- 
based courses provided similar or even higher results than traditional courses. A few 
studies found opposite results. In terms of satisfaction, web courses appeared to perform 
well, but there was some variation in the results. There may be additional factors such as 
experience and motivation that are influencing the results. The following study attempted 
to replicate components (i.e. predictor variables and outcome measures) of the previous 
studies in order to further address these issues and possibly provide some contributing 
evidence as to the overall effectiveness of online courses. 


Methodology 

Design 

For the comparison process, we conducted a study that spanned two academic years and 
four sections of a junior level course (Principles of Marketing). We compared traditional 
lecture style courses and web-based delivery courses during the study, which was similar 
in design to previous studies (Priluck, 2004). Our goal was to assess whether there was a 
difference in effectiveness between the two delivery formats. We defined effectiveness as 
the following: 

1. Learning Outcomes 

1.1. Final Exam 

1.2. Semester Project 
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1.3. Final Grade in Course 


2. Overall Course Satisfaction 

2.1. With Course 

2.2. With Instructor 

We made no special announcements that one section would be web-based until the first 
week of class. This helped to reduce self-selection bias. In the case of a strong objection 
by a student to the web format, then the student would have been allowed to switch 
sections. There were no instances in which this was the case. 

The traditional section of the course was conducted in a traditional “brick and mortar” 
environment with moderate utilization of technology such as, power- point presentations, 
TV/VCRs, overhead projectors, etc. It was essentially a lecture based format in a 
classroom setting that met twice a week. 

The web-based section was conducted via the Internet, which meant that all course 
lectures and materials were delivered via web-pages. We looked at a variety of studies 
(Kaynama and Keesling, 2000) in order to develop the class with careful design 
considerations (Taylor et ah, 2004). This used a modular format which presented 
information and incorporated learning activities on a weekly basis that was relevant and 
comparable to information being presented in the lecture-based sections (Su, 2005). 

In order to control for certain elements and measure the difference in others, we 
maximized the number of design elements that were similar. For instance, the instructor, 
the book, exams, assignments, projects, and material coverage were the same across all 
sections. The elements that were different included issues like delivery method and 
instructor contact. 


The Journal of Educators Online, Volume 4, Number 2, July 2007 


7 


Predictor Variables 

At the beginning of the semester, the predictor variables were measured across each 
section. The predictor variables are as follows: 

• GPA: Student grade point averages were taken from official University records. 

• Academic Level: Student academic level was taken from official class rosters. 
Categorized as Freshman, Sophomore, Junior or Senior (no graduate students 
enrolled in these undergraduate classes). 

• Web Based Experiences in classroom settings: Information was taken from 
pretests, indicating student self-reported experience with prior web-based 
classroom settings. 

• Perceived Knowledge: Information was taken from pretests, indicating student 
self-reported prior perceived knowledge with the Internet. 

• Course Format: This is not a measured variable, but it may influence the 
dependent variables. 

Based on other studies (Sweeney and Ingram, 2001), the authors felt that these predictor 
variables would all be important in helping to predict student performance in these 
classes. 

The variables were measured utilizing standard demographic questions and a series of 
questions for assessing experience and knowledge. For instance, experience questions 
were literally based on a series of yes/no questions that assessed whether they had 
participated in a particular online activity. The perceived knowledge questions were 
based on a series of questions that utilized a 5 point Likert-scale, which assessed the 
degree to which the respondent believed they had knowledge of the specific issue (Berger 
and Topal, 2001). 

Dependent Variables 

At the end of each semester, the dependent variables were measured, based on the 
following variable descriptions: 
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• Learning Outcomes: This information was taken from the various assignments 
that students submitted for the courses, such as the final exam, the semester 
project and the overall final grade. 

• Course Satisfaction: Information was taken from the student evaluations of the 
course at the end of the semesters. 


Both of these dependent variables were felt to be important by the researchers because 
they would help to explain the outcomes from the students at the end of the semesters. 

The measurement procedure for learning achievement was fairly straightforward as it was 
based on actual scores that were achieved by the students. While course satisfaction was 
measured utilizing a series of questions with Likert type scales ranging from “strongly 
agree” to “strongly disagree”. These questions are illustrated in table 3. 

Study 1 

Data for study 1 was collected from two sections of a Principles of Marketing course at a 
private Southeastern University. This study was funded via a Title III Strengthening 
Institutions Program grant {Achieving Excellence in Teaching & Learning.) The 
demographic and predictor variables are described in detail in Table 1, below: 


TABLE 3: Demographic and Predictor Variables for Study 1 


Study 1 


Traditional 

Online 

Sig-Diff 

Enrollment 


31 

17 

N/A 

Average GPA 


3.12 

3.08 

.093 

Gender: 

Male 

15 

6 

N/A 


Eemale 

16 

11 

N/A 

Academic Level 

Ereshman 

0 

0 

N/A 

Sophomore 

5 

1 

N/A 

Junior 

18 

9 

N/A 

Senior 

7 

7 

N/A 

Web-based 

Experiences 

Perceived Knowledge of Web 
Based Courses 

2.90 

3.07 

.078 

Experience in Online Course 

1.98 

1.93 

.181 

Experience with Online Course 
Components 

1.89 

2.00 

.074 
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As illustrated in table 3, the demographic and predictor variables have insignificant 
differences between the groups. This is important for establishing predictive validity as 
we examine the dependent variables. In terms of GPA’s, we found that there was no 
significant difference in GPA’s between the two sections (i.e. 3.12 and 3.08). Therefore, 
we assumed both sections started with similar academic backgrounds and potential for 
learning. The only variable with a notable difference was group size, because the online 
group was smaller than the traditional class. 

Study 2 

Data for study 2 was collected from two sections of a Principles of Marketing course at a 
second private Southeastern University. The study was conducted utilizing the same 
methodology and instruments as described in study 1 . The primary purpose of the study 
was to provide validation for study 1, and provide a longitudinal investigation of the 
issues. The demographic and predictor variables are described in detail in Table 4, below: 


TABLE 4: Demographic and Predictor Variables for Study 2 


Study 2 


Traditional 

Online 

Sig-Dif 

Enrollment 


35 

34 

N/A 

Average GPA 


3.25 

3.18 

.084 

Gender: 

Male 

15 

16 

N/A 


Eemale 

20 

18 

N/A 

Academic Level 

Eresbman 

0 

0 

N/A 

Sophomore 

4 

5 

N/A 

Junior 

20 

21 

N/A 

Senior 

11 

9 

N/A 

Web-based 

Experiences 

Perceived Knowledge of Web Based 
Courses 

3.10 

3.06 

.067 

Experience in Online Course 

2.07 

2.13 

.089 

Experience with Online Course 
Components 

3.15 

3.23 

.071 


Similar to study 1, we found very little difference between the groups within the total 
sample. We were also pleased to have similar group sizes. There was an interesting 
significant difference between study 1 and study 2, in that the sample from study 2 had a 
significantly greater amount of experience with online course components. 
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Results and Discussion 


The following results compare both the traditional course and the online course utilizing 
the pre-test and post-tests. The implications of each question are discussed. The statistical 
procedures utilized involved frequency analysis, means analysis, and a Mest to assess 
significant differences between means. 


TABLE 5: Dependent Variables 



Study 1 

Study 2 

Trad. 

Online 

Sig. 

Dif. 

Trad. 

Online 

Sig. 

Dif. 

Learning 

Achievement 

Final Exam 

76.4% 

75.3% 

.091 

79.3% 

78.4% 

.084 

Final Project 

85% 

84% 

.084 

86% 

84% 

.067 

Final Grade GPA 

3 

3.12 

.078 

3.15 

3.05 

.082 

Satisfaction 

Ql: Excellent Teacher 

1.13 

1.18 

.754 

1.12 

1.19 

.069 

Q2: Overall, I learned a great deal 
in this course. 

1.22 

1.25 

.086 

1.21 

1.25 

.079 

Q3: 1 would enjoy taking another 
course with this instructor. 

1.09 

1.12 

.082 

1.08 

1.16 

.047 

Q4: The instructor keeps students 
interested and motivated. 

1.22 

1.25 

.086 

1.17 

1.26 

.046 

Q5; The instructor creates a 
classroom environment that 
encourages students to learn. 

1.17 

1.25 

.043 

1.14 

1.29 

.039 

Q6; The instructor presents course 
materials in a clear and organized 
manner. 

1.09 

1.12 

.082 

1.05 

1.15 

.063 

Q7 : The instructor utilizes time 
effectively and appropriately. 

1.22 

1.25 

.086 

1.13 

1.19 

.061 

Q8; Overall, I consider this course 
to be very important for my 
education. 

1.13 

1.18 

.754 

1.17 

1.18 

.127 


* There was no significant difference for any measure between the studies. 
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Learning Outcomes 

In terms of learning outcomes, we assessed results on the final exam, results on the 
semester project, and overall results in the course. In terms of course performance, we 
found no significant differences between sections across all four groups. The final exam 
consisted of 50 multiple choice questions and 3 of 4 short- answer essays. The semester 
project entailed the development of a marketing plan for a product or service of the 
students’ choice. The online class tended to have projects that focused on some type of 
business entity with an online perspective. The overall course grade was a combination of 
the exams and the semester project. The results indicate that there was relatively little 
difference in learning achievement between groups or across studies. This is an indication 
that the online students performed equally well with the students in the traditional course. 
Although grades were similar, we did find that the drop rate was higher for the online 
sections. Even though the sample of those students that dropped was not large enough for 
appropriate statistical analysis, we can say that those who dropped were younger, had less 
experience, and had lower GPAs. 

Course Satisfaction 

In order to measure general course satisfaction, we utilized the standardized course 
evaluation forms used by one of the Schools of Business across all sections used in this 
study. There are 8 questions which are evaluations of the course and instructor, which 
utilize a 5 point Likert type scale ranging from “strongly agree” to “strongly disagree”. 
The actual questions are illustrated in Table 5. The general results indicate that there is no 
significant difference between the two studies. But there are statistically significant 
differences and visually notable differences between sections. It seems that the online 
section tends to have higher ratings (lower being better) than the traditional section. This 
is an indicator that the students are satisfied with convenience and delivery mechanism, 
but they tend to miss the personal interaction with the instructor. This is generally the 
number one cause for higher ratings across the board, and this is due to the fact that the 
online environment is simply less personal. 
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Regression 

In addition to the measurement of differences between groups, we assessed the degree to 
which the predictor variables actually predicted the dependent variables. This was 
important because it would help to identify which variables were important for success in 
both of these classroom environments. The results of the hierarchical regression for 
predicting learning achievement (Table 6) and course satisfaction (Table 7) for both 
studies are illustrated with significant results in bold. 


TABLE 6: Regression for Learning Achievement: Study 1 and Study 2 


Predictor 

Variables 

Study 1 

Study 2 


Change 

Beta 

R- 

R- 

Change 

Beta 

GPA 

.223 

.223 

.360 

.228 

.228 

.364 

Course Format 

.227 

.004 


.235 

.007 


Academic Level 

.247 

.02 

.144 

.250 

.015 

.135 

Knowledge 

.264 

.017 

.136 

.268 

.018 

.138 

Experience 

.270 

.003 


.280 

.012 

.121 


In study 1, the variables explained about 27% of the variance in learning achievement, 
while 28% was explained in study 2. GPA provided the most significant explanation of 
success in terms of learning achievement. A subsequent means analysis regarding GPA 
indicated that those with higher initial GPAs tended to perform better on the learning 
achievement variables. Course format had an insignificant contribution to learning 
achievements, which was a positive result for our study. This indicates that students learn 
equally in both formats. Both academic level and perceived knowledge were significant 
contributors to the equation, therefore indicating that upper level students with higher 
perceived knowledge tend to due better in either course. Finally, it was interesting to note 
that experience was only influential in study 2, as the students tended to have more 
experience in online environments. This seems logical as students tend to perform better 
in an environment in which they have greater experience. 
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TABLE 7: Regression for Course Satisfaction: Study 1 and Study 2 


Predictor 

Variables 

Study 1 

Study 2 

r" 

Change 

Beta 

R- 

R- 

Change 

Beta 

Course Format 

.245 

.145 

.376 

.248 

.248 

.389 

Academic Level 

.258 

.013 

.130 

.268 

.020 

.139 

GPA 

.261 

.003 


.272 

.004 


Knowledge 

.281 

.020 

.138 

.280 

.012 

.129 

Experience 

.287 

.006 


.295 

.015 

.132 


In study 1, the variables explained about 28% of the variance in course satisfaction, while 
29% was explained in study 2. Course format provided the most significant explanation 
of variance for course satisfaction. There were several similarities in the regression 
equations, indicating that both academic level and perceived knowledge were also 
significant contributors to the explanation of satisfaction, and that there was a difference 
in the importance of experience between study 1 and study 2. The major difference 
between the equations was that GPA had relatively little impact on course satisfaction, 
which was confirmed in a subsequent group means analysis. 


In total, the results contribute to the discussion by showing that online formats can 
deliver comparable learning outcomes as that of traditional courses. Although, motivation 
is not a factor that we measured directly, it does appear to be an influential and 
interesting factor that appears to influence the degree of success that students enjoy in the 
online environment. Satisfaction is a more difficult element to gage as we found that 
students are generally satisfied with the convenience and format of online courses, but 
they generally want more interaction with the instructor. Additional strategies and 
techniques need to be developed that enhance the instructor-student interaction scenario. 
For instance, it may be necessary to set-up appointment times for online discussions or 
set-up course blogs. It terms of technology skills, this appears to have been a more 
important issue in previous studies. Our results indicate that students are adequately 
prepared for the online course environment in today’s technology driven atmosphere. 
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Conclusions 


Overall, the outcomes of this study lend strong support to the development and utilization 
of online technologies in the delivery of course materials. The following items represent 
the top 5 outcomes that we found while conducting this research: 

1 . Similar course objectives and goals are achieved in the online environment. 

2 . The same degree of learning outcomes was achieved in the online 
environment. 

3. Course satisfaction was comparable to the traditional section, although a little 
lower, but it was seemingly an acceptable trade-off for convenience. 

4 . Students are comfortable with the technology and delivery environment. 

5 . The primary negative element is the lack of personal interaction. 

The overall results of study 1 and study 2 help to substantiate some previous studies, and 
possibly rebuke the results of others. It seems we had similar results in terms of 
satisfaction (Maki and Maki, 2002), in that web-based sections tended to have lower 
satisfaction ratings. Our results differed from Wang and Newlin (2000), since we did not 
find a significant difference in learning achievement between groups. 

The use of online courses can be very beneficial for the university community. The use of 
technology has many benefits in the learning environment. Essentially, online technology 
makes education available at any time and any place. Our research indicates that the 
online environment provides a comprehensive and comparable learning environment. Yet, 
as with any endeavor, there are pros and cons, which must be taken into account for a 
balanced strategy. As found in other marketing studies (Eastman and Swift, 2001), 
students enjoy the convenience provided by the online environment, but they miss the 
personal interaction. Thus, the online environment does not provide a warm and caring 
environment, and this presents a significant tradeoff with the traditional classroom 
environment. Yet, the integration of some online courses can be beneficial to both the 
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students and a university. For instance, online courses can be a productive strategy for 
driving course enrollments during the summer. The summer semester has traditionally 
shown lower enrollment figures, but with the integration of more online courses during 
the summer a University could expect to see higher enrollments. This is due to the 
convenience factor, because students can take courses from anywhere (including home) 
Obviously, there are also other benefits to the technology associated with online 
environment, in terms of supplementing the traditional course environment. This 
technology can provide a valuable information portal in a variety of traditional course 
settings. These results may indicate that a hybrid type of course could provide the ideal 
balance between interaction and convenience. 


Implications for Future Research 

The results of these studies have provided answers to some questions, while uncovering 
other issues that need additional research. For instance, it seems that a hybrid type of 
course that combines the benefits of an online environment with a traditional course may 
be very effective. Future studies may want to compare web-only, hybrid, and traditional 
courses to assess and directly compare the effectiveness of each. Another issue that might 
be important in both the web-only and hybrid learning environments is the concept of 
motivation. While we did not measure it in this study, it may be an important predictor 
variable for learning outcomes. 

Finally, the technology and potential for online courses changes rapidly, therefore it is 
imperative to conduct ongoing research. A continuous longitudinal research approach 
would help to facilitate the continued development and refinement of online courses to 
ensure that various benchmarks are being achieved. In terms of benchmarks, it would be 
in the best interest of a university to establish a series of benchmarks, evaluation 
guidelines, online course strategy, etc., as we can expect the availability of online courses 
in the education community to continue to grow. This growth should be managed and 
coordinated centrally. 
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